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as  a  Certified  Defense  Financial  Manager  with  Acquisition  Specialty  (CDFM-A).  [wrfast@nps.edu] 

Abstract 

On  May  22,  2009,  the  President  signed  into  law  the  Weapon  System  Acquisition  Reform  Act 
of  2009  (WSARA).  An  important  feature  of  WSARA  is  the  requirement  for  all  Major  Defense 
Acquisition  Programs  (MDAPs)  to  conduct  competitive  prototyping  prior  to  the  Milestone  B 
development  decision.  These  prototypes  must  be  demonstrated  in  a  relevant  environment  to 
provide  the  milestone  decision  authority  (MDA)  with  an  assessment  of  their  level  of 
technology  maturity.  Competitive  prototyping  in  this  early  phase  can  also  identify  program  risk 
and  help  the  MDA  decide  if  there  is  a  good  match  between  the  customer’s  needs  and  the 
available  resource — technology,  design,  time,  and  funding.  However,  if  the  MDA  determines 
that  there  is  little  or  no  benefit,  competitive  prototyping  can  be  waived.  The  rationale  behind 
such  a  waiver  must  be  provided  to  the  Government  Accountability  Office  (GAO).  The  WSARA 
competitive  prototyping  requirement  has  now  been  in  effect  for  nearly  seven  years.  There  is  a 
considerable  amount  of  data  on  the  costs  and  benefits  of  early  competitive  prototyping 
efforts.  In  addition,  the  GAO  has  published  numerous  opinions  regarding  MDA  waivers  of 
competitive  prototyping.  This  research  analyzes  MDAP  data  from  the  2013,  2014,  and  2015 
Annual  Reports  on  the  Performance  of  the  Defense  Acquisition  System,  the  Defense 
Acquisition  Management  Information  Retrieval  (DAMIR),  Budget  Exhibits  that  accompany  the 
President’s  annual  budget  request,  the  Director  of  Operational  Test  and  Evaluation  (DOT&E), 
and  the  GAO.  This  research  finds  that  early  competitive  prototyping  has  received  only  partial 
implementation  for  pre-MDAP  and  MDAP  programs.  However,  when  implemented,  early 
competitive  prototyping  has  reduced  Program  Acquisition  Unit  Cost  (PAUC)  and  has  reduced 
technology  risk. 

Research  Questions 


1 .  What  MDAPs,  entering  the  EMD  phase  after  May  2009,  have  received  a  waiver  from 
competitive  prototyping  prior  to  Milestone  B?  What  were  the  most  common  reasons 
for  these  waivers?  Did  the  GAO  concur  with  these  waivers  and,  if  not,  why  not? 

2.  What  MDAPs,  entering  the  EMD  phase  after  May  2009,  have  conducted  competitive 
prototyping  prior  to  Milestone  B?  For  these  MDAPs,  what  were  the  results  of 
competitive  prototyping  in  terms  of  technology  maturity,  design  risk  reduction, 
development  cycle  time,  and  cost  estimate/funding  stability,  and  system 
performance? 

3.  Based  upon  the  answers  to  research  questions  1  and  2,  has  the  WSARA  early 
competitive  prototyping  requirement  helped  reduce  MDAP  Program  Acquisition  Unit 
Cost  (PAUC)  and  technology  risk? 


Background 

Section  203  of  the  Weapon  System  Acquisition  Reform  Act  (WSARA)  of  2009 
requires  that  the  acquisition  strategies  for  Major  Defense  Acquisition  Programs  (MDAPs) 
provide  for  competitive  prototyping  prior  to  the  Milestone  B  approval.  The  Milestone 
Decision  Authority  (MDA)  may  waive  the  competitive  prototyping  requirement  only  on  the 
basis  that 
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a)  the  cost  of  producing  competitive  prototypes  exceeds  the  expected  life- 
cycle  benefits  (in  constant  dollars)  of  producing  such  prototypes,  including  the 
benefits  of  improved  performance  and  increased  technological  and  design 
maturity  that  may  be  achieved  through  competitive  prototyping;  or  b)  on  the 
basis  that,  but  for  such  waiver,  the  Department  would  be  unable  to  meet 
critical  national  security  objectives.  (WSARA,  2009,  §  203) 

The  language  of  Section  203  was  later  modified  to  clarify  that  prototypes  may  be  acquired 
from  commercial,  government,  or  academic  sources  and,  if  prototyping  of  the  system  is  not 
feasible,  prototypes  may  be  acquired  for  critical  subsystems  (WSARA,  2009,  §  813). 

To  better  understand  this  statute,  several  terms  must  be  defined.  First,  there  are 
generally  two  types  of  prototyping  incorporated  into  acquisition  strategies.  Early  prototyping 
is  done  in  the  Technology  Maturation  and  Risk  Reduction  (TMRR)  phase  of  the  program, 
prior  to  Milestone  B.  Prototyping  does  occur  later,  after  Milestone  B,  when  Engineering 
Design  Models  (EDM)  are  built  in  the  Engineering  and  Manufacturing  Development  (EMD) 
phase  of  the  program.  However,  the  intent  of  the  WSARA  of  2009  is  for  MDAPs  to  have 
early  prototypes,  prior  to  approval  of  Milestone  B. 

Competitive  prototyping  involves  two  or  more  contractors  developing  prototypes  and 
verifying  (through  testing  or  demonstration)  that  they  meet  contractual  requirements.  Ideally, 
after  the  contractor  builds  and  demonstrates  or  tests  the  prototype,  the  government  and 
contractor  conduct  a  Preliminary  Design  Review  (PDR).  The  purpose  of  the  PDR  is  to 
assess  the  results  of  the  prototyping  effort  and  to  conduct  cost,  schedule,  and  performance 
trades  to  meet  user  requirements,  schedule,  and  resources.  For  example,  some  operational 
requirements  may  have  to  be  deferred  to  a  later  increment  of  the  program  because  certain 
critical  technologies  were  determined  as  immature  while  testing  a  competitive  prototype. 

A  Major  Defense  Acquisition  Program  (MDAP)  is  a  program  that  expects  to  expend 
more  than  $480  million  in  Research,  Development,  Test,  and  Evaluation  (RDT&E;  FY2014 
constant  dollars)  or  $2.79  billion  in  procurement  (FY2014  constant  dollars)  for  all  of  its 
increments  (10  U.S.C.  §  2430).  However,  at  the  time  that  the  WSARA  of  2009  was  enacted, 
the  RDT&E  expected  expenditure  threshold  for  an  MDAP  was  $365  million  and  the 
procurement  expected  expenditure  threshold  was  $2.19  billion  (FY2000  constant  dollars). 

When  a  program  is  designated  as  an  MDAP,  the  Milestone  Decision  Authority  (MDA) 
is  either  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (who  is 
also  the  Defense  Acquisition  Executive,  DAE)  or  the  designated  Component  Acquisition 
Executive  (CAE).  The  MDA  determines  when  programs  move  from  one  acquisition  phase  to 
the  next  by  reviewing  program  accomplishments  at  the  various  milestone  reviews 
throughout  the  acquisition  framework  (DoDI,  2015a). 

Since  the  WSARA  of  2009  was  enacted,  DoD  Instruction  5000.02,  Operation  of  the 
Defense  Acquisition  System,  has  changed  three  times.  To  accommodate  these  changes, 
only  the  current  names  for  acquisition  phases,  reviews,  and  capability  documents  are  used 
throughout  this  paper. 

Literature  Review 

The  only  published  research  on  the  implementation  of  early  prototyping  by  MDAPs 
since  2009  are  the  Government  Accountability  Office  (GAO)  annual  Assessments  of 
Selected  Weapon  Programs.  Recently,  these  assessments  have  been  based  upon  an 
electronic  questionnaire  sent  to  selected  program  management  offices.  GAO  pre-tests  their 
questionnaire  to  ensure  that  the  program  offices  of  each  Service  understand  the  terminology 
used  in  the  questionnaire.  One  set  of  questions  in  the  survey  asks  if  the  program  has 
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attained  knowledge  that  all  of  its  critical  technologies  work  in  a  relevant  environment  and  in 
a  realistic  environment.  Technologies  that  work  in  a  relevant  environment  would  be 
assessed  at  Technology  Readiness  Level  (TRL)  6.  Technologies  proven  to  work  in  a 
realistic  environment  would  be  assessed  at  TRL  7.  However,  neither  of  these  questions 
specifically  ask  if  the  technologies  were  demonstrated  on  early  competitive  prototypes.  The 
program  offices  could  have  demonstrated  the  technology  on  components,  subsystems,  or 
even  on  a  single  prototype,  rather  than  on  competitive  prototypes.  Therefore,  the  results  of 
the  electronic  questionnaire  are  inconclusive  as  to  whether  early  competitive  prototyping 
was  actually  used. 

The  program  offices  most  certainly  answer  these  two  GAO  questions  based  upon 
their  interpretation  of  the  early  competitive  prototyping  requirement.  For  example,  the  F-22 
Increment  3.2B  Modernization  (F-22  Inc  3.2B  Mod)  program  office  reported  to  the  GAO  that 
all  critical  technologies  were  demonstrated  in  a  relevant  environment  prior  to  Milestone  B. 
However,  there  is  no  evidence  in  any  independent  source  documents  that  confirms  that 
competitive  prototypes  were  used  for  that  technology  assessment  or  that  the  critical 
technologies  were  tested  on  even  a  single  prototype.  Thus,  based  upon  the  GAO  report,  we 
can  only  assume  that  the  program  office  tested  the  critical  technologies  on  components  or 
subsystems,  but  not  using  an  integrated  system  prototype.  Similar  assumptions  must  be 
made  for  the  K-46A  Tanker  Modernization  program  and  the  Global  Positioning  System 
(GPS)  Next  Generation  Operational  Control  System  (OCX) — no  competitive  prototypes  or 
even  a  single  prototype  was  developed  and  tested. 

The  other  disconnect  in  the  GAO  questionnaire  results  involves  programs  that  have 
had  competitive  and  single  prototype  waivers.  For  example,  the  Combat  Rescue  Helicopter 
(CRH)  program  received  a  waiver  from  early  prototyping.  In  the  business  case  analysis  for 
that  waiver,  the  CRH  program  office  reports  that  critical  technologies  are  mature.  Yet,  in 
response  to  the  GAO  questionnaire,  the  CRH  program  office  says  that  information  is  not 
available  as  to  whether  critical  technologies  have  been  demonstrated  in  a  relevant  or  a 
realistic  environment.  A  different  kind  of  disconnect  is  seen  in  the  Armored  Multi-Purpose 
Vehicle  (AMPV)  and  the  VH-92A  Presidential  Helicopter  Replacement  (VH-92A)  programs. 
In  these  two  programs,  the  program  offices  report  the  demonstration  of  critical  technologies 
in  a  relevant  environment),  yet  both  programs  received  prototyping  waivers  (and  the  GAO 
clearly  states  this  in  the  assessment  of  the  AMPV  program).  So,  these  two  program  offices 
must  be  referring  to  component  or  subsystem  prototyping  (GAO,  2015a). 

Methodology 

This  research  effort  used  only  objective  evidence,  not  questionnaires  filled  out  by 
program  offices  that  could  be  biased  or  open  to  interpretation.  First,  all  new  MDAPs  entering 
the  EMD  phase  since  2009  were  identified  by  extracting  new  programs  first  reporting  a 
development  baseline  in  the  Selected  Acquisition  Report  (SAR)  summary  tables.  SARs  on 
all  MDAPs  are  prepared  by  the  Department  of  Defense  (DoD)  and  sent  to  the  Congress 
annually.  Second,  pre-MDAP  programs  that  had  early  competitive  prototyping  efforts  since 
2009  were  identified  from  the  data  reported  in  the  Defense  Acquisition  Management 
Information  Retrieval  (DAMIR). 

Several  new  MDAPs  did  not  have  an  early  prototyping  effort  because  they  were 
granted  a  waiver  by  the  Milestone  Decision  Authority  (MDA).  In  accordance  with  the 
WSARA  of  2009,  the  MDA’s  rationale  for  granting  a  waiver  from  competitive  prototyping 
must  be  reported  to  the  Comptroller  General  of  the  United  States.  The  GAO  assesses  this 
rationale  and  sends  a  written  report  of  their  findings  to  the  congressional  defense 
committees.  Copies  of  these  waiver  assessments  are  publicly  available  on  the  GAO 
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website.  All  GAO  assessments  of  competitive  prototyping  waivers  since  2009  were 
reviewed,  to  include  the  rationale  for  the  waiver.  For  example,  was  the  waiver  based  upon 
National  Security  urgency?  Or,  was  the  waiver  based  upon  a  cost  benefit  analysis  (CBA) 
where  the  costs  of  prototyping  outweighed  the  dollarized  benefits  in  constant  dollars?  If  the 
waiver  was  based  upon  a  CBA,  then  answers  were  sought  for  these  additional  questions: 

•  What  were  the  costs  and  benefits  cited  in  the  CBA?  Were  the  costs  and 
benefits  properly  determined  and  compared? 

•  What  type  of  contract  was  subsequently  used  in  the  EMD  phase?  If  a  firm 
fixed  price-type  contract  was  used  in  the  EMD  phase,  it  may  be  that  there 
was  little  to  no  technical  risk.  That  would  validate  the  decision  to  waive  early 
prototyping. 

•  What  was  the  baseline  cost  estimate  for  Program  Acquisition  Cost  (in 
constant  dollars),  and  what  was  the  quantity  established  at  the  development 
baseline  (i.e. ,  program  initiation  and  the  first  SAR  to  Congress)?  What  is  the 
current  Program  Acquisition  Cost  estimate  (in  constant  dollars)  and  what  is 
the  quantity  reported  in  the  2015  SAR?  Is  there  evidence  of  cost  growth  or 
cost  reduction  between  the  Program  Acquisition  Unit  Cost  (PAUC)  at  the 
development  baseline  and  the  current  PAUC  as  of  the  2015  SAR? 

For  each  MDAP  that  did  not  have  a  competitive  prototyping  waiver,  the  type  of  early 
prototyping  done  prior  to  the  Engineering  and  Manufacturing  Development  (EMD)  phase 
was  identified  (i.e.,  subsystem/component  prototypes,  a  single  system  prototype,  and 
competitive  prototypes).  Then,  these  questions  about  those  prototyping  efforts  were 
answered: 


•  What  early  prototypes  were  built,  and  how  were  they  tested?  Whenever 
available,  independent  reports  of  test  efforts  by  the  DoD  Director  of  Systems 
Engineering,  the  DoD  Director  of  Developmental  Testing,  and  the  DoD 
Director  of  Operational  Testing  were  used. 

•  What  technology  readiness  levels  (TRL)  and  manufacturing  readiness  levels 
(MRL)  resulted  from  the  prototyping  effort?  Whenever  available,  independent 
TRL/MRL  assessments  by  the  DoD  Director  of  Systems  Engineering  were 
used. 

•  What  type  of  contract  was  used  in  the  TMRR  phase  for  the  prototyping  effort, 
and  what  type  of  contract  was  subsequently  used  in  the  EMD  phase?  When 
available,  contract  types  were  taken  from  Form  R-3  Budget  Exhibits.  The 
point  of  asking  this  question  is  to  understand  the  level  of  technical  risk, 
especially  after  early  prototyping.  For  example,  if  a  Cost  Plus  Fixed  Fee 
(CPFF)  contract  was  used  in  the  EMD  phase,  one  might  question  whether  the 
early  prototyping  effort  revealed  the  technical  maturity  of  the  design. 

•  What  was  the  baseline  cost  estimate  for  Program  Acquisition  Cost  (in 
constant  dollars),  and  what  was  the  quantity  established  at  the  development 
baseline  (i.e.,  program  initiation  and  the  first  SAR  to  Congress)?  What  is  the 
current  Program  Acquisition  Cost  estimate  (in  constant  dollars)  and  what  is 
the  quantity  reported  in  the  2015  SAR?  Is  there  evidence  of  cost  growth  or 
cost  reduction  between  the  Program  Acquisition  Unit  Cost  (PAUC)  at  the 
development  baseline  and  the  current  PAUC  as  of  the  2015  SAR? 
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Findings 

Since  the  enactment  of  the  WSARA  of  2009,  there  have  been  about  28  pre-MDAP  or 
MDAP  programs  subject  to  statutory  competitive  prototyping  requirements.  These  programs 
can  be  placed  into  three  categories  as  shown  in  Table  1. 


Table  1.  Pre-MDAPs  and  MDAPs  Subject  to  Statutory  Prototyping  Since  2009 


Waiver 

No  Prototyping 

Prototyping 

•  AMPV 

•  B61  Mod  12LEPTKA 

•  AM  DR 

•  B-2  DMS  Mod* 

•  F-22  Inc  3.2B  Mod 

•  Chem  Demil-ACWA 

•  CRH 

•  FAB-T  FET 

•  EPS 

•  EPS  CAPS 

•  GPS  OCX 

•  F-35  Aircraft 

•  F-15  EPAWSS* 

•  ICBM  Fuze  Mod 

•  F-35  Engine 

•  IFPC  Inc  2  Blk  1 

•  KC-46A 

•  JAGM 

•  T-AO(X)* 

•  LCS  MM 

•  JLTV 

•  VH-92A 

•  LCS 

•  NGJ* 

•  SDB II 

•  SF 

•  SSC 

*pre-MDAP 

•  3DELRR* 

The  next  three  sections  summarize  the  common  characteristics  of  the  programs  with 
prototyping  waivers,  no  prototyping,  or  prototyping  efforts. 

Waivers 

Since  the  enactment  of  the  WSARA  of  2009,  8  pre-MDAP  or  MDAP  programs 
received  a  waiver  from  early  prototyping  (see  Table  1 ).  A  description  of  each  of  these 
programs  and  their  cost  benefit  analyses  are  at  Appendix  A.  The  findings  follow: 

•  Rationale  Behind  Waivers.  Seven  (7)  of  these  waivers  were  based  upon  the 
rationale  that  the  cost  of  producing  the  competitive  prototypes  outweighed  the 
life-cycle  benefits  of  the  prototyping  effort.  The  waiver  for  EPS  CAPS  was 
based  upon  the  rationale  that  the  cost  of  prototyping  outweighed  the  benefits 
and  that  the  delay  caused  by  prototyping  would  jeopardize  National  Security. 

•  GAO’s  Assessments.  In  most  cases,  the  GAO’s  assessments  found  that  the 
rationale  behind  the  prototyping  waivers  was  consistent  with  the  intent  of  the 
WSARA  of  2009.  However,  in  assessing  the  waivers  for  AMPV,  EPS  CAPS, 
B-2  DMS-M,  EPS  CAPS,  and  VH-92A,  the  GOA  questioned  whether  a 
sufficient  number  of  prototyping  alternatives  had  been  analyzed  or  criticized 
the  effort  (or  lack  of  effort)  to  dollarize  prototyping  benefits. 

•  Some  Waivers  Supported  by  Acquisition  Strategy.  Acquisition  strategies 
to  reduce  technology  risk  by  using  only  mature  technologies  and/or  by 
reducing  user  requirements  often  obviated  the  need  for  early  risk  reduction 
prototyping.  This  was  the  case  with  the  waivers  for  AMPV,  CRH,  F-15 
EPAWSS,  IFPC  Inc  2  Blk  1,  T-AO(X),  and  VH-92A. 

•  Cost  Reimbursement  Contract  for  EMD  Questioned.  The  GAO’s 
assessments  of  the  prototyping  waivers  always  questioned  whenever  a  cost 
reimbursement-type  contract  was  used  or  proposed  for  the  EMD  phase.  Their 
rationale  is  that  early  prototyping  may  reduce  technology  risk,  permitting  the 
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use  of  a  fixed  price-type  contract  for  the  EMD  phase.  Two  areas  of  risk  often 
identified  as  justifying  the  use  of  cost  reimbursement-type  contracts  were 
software  development  and  integration.  This  was  the  case  with  the  waivers  for 
AMPV,  B-2  DMS  Mod,  and  EPS  CAPS. 

•  Schedule  Impacts  Not  Quantified  in  Dollars.  Schedule  impacts  due  to 
prototyping  were  discussed  in  6  of  the  8  waivers.  Depending  on  the  number 
of  competing  contractors,  the  delays  due  to  prototyping  ranged  from  6 
months  to  60  months.  However,  none  of  the  schedule  impacts  appear  to  have 
been  added  into  the  cost  of  the  prototyping  effort  for  direct  comparison  to  the 
benefits  in  dollars. 

•  BCA  Not  Posted  With  Waiver  Assessment.  The  business  case  analysis 
(BCA)  done  by  the  program  office  was  not  posted  along  with  the  GAO 
assessment  of  the  waiver.  Had  the  BCA  been  made  available,  it  might  have 
been  useful  in  answering  some  of  these  research  questions.  More 
importantly,  a  copy  of  the  BCA  posted  with  the  GAO  assessment  of  that  BCA 
could  be  a  good  teaching  tool  for  the  Defense  Acquisition  Workforce. 

•  ASD  Acquisition  Comments  on  GAO  Assessments.  The  Assistant 
Secretary  of  Defense  for  Acquisition  commented  on  three  of  the  GAO’s 
waiver  assessment  reports  that  “competitive  prototyping  needs  to  be  tailored 
to  the  needs  and  risks  of  each  specific  program,  balanced  with  any  potential 
adverse  cost  and  schedule  impact.” This  comment  may  indicate  that  the  ASD 
Acquisition  has  a  different  interpretation  of  the  early  prototyping  requirement 
than  does  the  GAO. 


No  Prototyping 

Since  the  enactment  of  the  WSARA  of  2009,  seven  (7)  pre-MDAP  or  MDAP 
programs  did  not  conduct  any  early  prototyping  (see  Table  1).  A  description  of  each  of  these 
programs  is  at  Appendix  B.  The  findings  follow: 


•  Programs  Started  in  EMD  Phase.  All  7  of  these  programs  may  have 
assumed  that  just  because  they  bypassed  the  TMRR  phase  and  started  in 
the  EMD  phase,  they  didn’t  have  to  do  competitive  prototyping.  However,  that 
assumption  is  clearly  not  the  intent  of  the  WSARA  of  2009  which  states  that 
competitive  prototyping  is  required  before  Milestone  B  (per  Section  203  of  the 
WSARA  of  2009).  All  of  these  programs  had  to  go  through  Milestone  B  before 
entering  the  EMD  phase.  In  addition,  prior  to  the  Milestone  B  decision,  the 
Milestone  Decision  Authority  must  certify,  to  the  Congress,  that  the 
technology  in  the  program  has  been  demonstrated  in  a  relevant  environment 
(TRL  6)  and  that  the  program  complies  with  all  relevant  policies,  regulations, 
and  directives  of  the  Department  of  Defense  (per  10  U.S.C.  §  2366b).  DoDI 
5000.02,  Operation  of  the  Defense  Acquisition  System,  has  required 
competitive  prototyping  since  2008. 

•  Reductions  in  PAUC.  When  the  Program  Acquisition  Unit  Cost  (PAUC)  in 
constant  dollars,  based  upon  the  development  (EMD)  baseline,  is  compared 
to  the  current  PAUC  in  constant  dollars,  based  upon  the  2015  Annual  SAR, 
four  (4)  of  the  programs  have  shown  a  reduction  in  PAUC.  The  B61  Mod  12 
LEP  TKA  program  has  the  largest  reduction  in  PAUC  at  -9.6%.  The  LCS  MM 
program  has  the  smallest  reduction  in  PAUC  at  -0.1%. 

•  Increases  in  PAUC.  When  the  Program  Acquisition  Unit  Cost  (PAUC)  in 
constant  dollars,  based  upon  the  development  (EMD)  baseline,  is  compared 
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to  the  current  PAUC  in  constant  dollars,  based  upon  the  2015  Annual  SAR, 
three  (3)  of  the  programs  have  shown  an  increase  in  PAUC.  The  GPS  OCX 
program  has  the  largest  increase  in  PAUC  at  +21.2%.  The  ICBM  Fuze  Mod 
program  has  the  smallest  increase  in  PAUC  at  +1.0%. 

•  Cost  Reimbursement  Development  (EMD)  Contracts.  Five  (5)  of  the 

programs  awarded  a  cost  reimbursement-type  contract  for  the  development 
(EMD)  phase.  Cost  Plus  Fixed  Fee  (CPFF)  contracts  were  awarded  for  ICBM 
Fuze  Mod  and  LCS  MM.  Cost  Plus  Award  Fee  (CPAF)  contracts  were 
awarded  for  GPS  OCX  and  LCS  MM.  Cost  Plus  Incentive  Fee  (CPIF) 
contracts  were  awarded  for  B61  Mod  12  LEP  TKA  and  F-22  Inc  3.2B  Mod. 
The  determination  of  the  contract  type  for  the  development  (EMD)  phase  of 
an  MDAP  is  the  responsibility  of  the  Milestone  Decision  Authority  (MDA).  If 
the  MDA  selects  a  cost  reimbursement-type  contract  for  the  development 
(EMD)  phase,  this  decision  has  to  be  justified  to  the  Congress.  In  the 
justification,  the  MDA  is  to  explain  why  the  technology  is  so  immature  that  a 
cost  reimbursement  contract  is  necessary  (DoD,  2015a). 

•  Identification  of  Critical  Technology  Elements.  Software  development  and 
system  integration  were  often  not  considered  as  critical  technology  elements 
that  could  be  assessed  and  matured  through  early  prototyping.  This  was  the 
case  with  F-22  Inc  3.2B  Mod,  GPS  OCX,  and  LCS  MM. 


Prototyping 

Since  the  enactment  of  the  WSARA  of  2009,  thirteen  (1 3)  pre-MDAP  or  MDAP 
programs  did  conduct  early  prototyping  (see  Table  1).  A  description  of  each  of  these 
programs  is  at  Appendix  C.  The  findings  follow: 


•  Types  of  Prototyping  Used.  All  13  of  the  programs  used  some  type  of 
prototyping  permitted  under  the  WSARA  of  2009.  One  (1)  program  (Chem 
Demil-ACWA)  prototyped  two  separate  components.  One  (1)  program  (SSC) 
used  a  subsystem  prototype.  Eleven  (11)  programs  (AMDR,  EPS,  F-35 
Aircraft,  F-35  Engine,  JAGM,  JLTV,  LCS,  NGJ,  SDB  II,  SF,  and  3DELRR) 
used  2  or  more  subsystem  or  competitive  prototypes. 

•  Reductions  in  PAUC.  When  the  Program  Acquisition  Unit  Cost  (PAUC)  in 
constant  dollars,  based  upon  the  development  (EMD)  baseline,  is  compared 
to  the  current  PAUC  in  constant  dollars,  based  upon  the  2015  Annual  SAR, 
eight  (8)  of  the  programs  have  shown  a  reduction  in  PAUC.  The  SDB  II 
program  has  the  largest  reduction  in  PAUC  at  -21.3%.  The  EPS  program  has 
the  smallest  reduction  in  PAUC  at  -0.6%. 

•  Increases  in  PAUC.  When  the  Program  Acquisition  Unit  Cost  (PAUC)  in 
constant  dollars,  based  upon  the  development  (EMD)  baseline,  is  compared 
to  the  current  PAUC  in  constant  dollars,  based  upon  the  2015  Annual  SAR, 
three  (3)  of  the  programs  have  shown  an  increase  in  PAUC.  The  Chem 
Demil-AWCA  program  has  the  largest  increase  in  PAUC  at  +3.0%.  The 
JAGM  program  has  the  smallest  increase  in  PAUC  at  +0.2%. 

•  Two  (2)  pre-MDAP  programs  (NGJ  and  3DELRR)  did  not  have  development 
(EMD)  baselines  from  which  to  determine  any  increase  or  decrease  in  PAUC. 

•  Fixed  Price  Development  (EMD)  Contracts.  Five  (5)  of  the  programs 
awarded  a  fixed  price-type  contract  for  the  development  (EMD)  phase.  Fixed 
Price  Incentive  (Firm  Target;  FPIF)  contracts  were  awarded  for  LCS,  SDB  II, 
SF,  and  SSC.  A  Firm  Fixed  Price  (FFP)  contract  was  awarded  for  JLTV.  Use 
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of  a  fixed  price-type  contract  is  an  indication  that  technology  is  mature  and 
cost  risk  has  been  reduced. 

•  Cost  Reimbursement  Development  (EMD)  Contracts.  Five  (5)  of  the 

programs  were  awarded  a  cost  reimbursement-type  contract  for  the 
development  (EMD)  phase.  Cost  Plus  Award  Fee  (CPAF)  contracts  were 
awarded  for  Chem  Demil-ACWA,  F-35  Aircraft,  and  F-35  Engine.  Cost  Plus 
Incentive  Fee  (CPIF)  contracts  were  awarded  for  AMDR,  Chem  Demil- 
ACWA,  EPS,  F-35  Aircraft,  and  F-35  Engine.  The  determination  of  the 
contract  type  for  the  development  (EMD)  phase  of  an  MDAP  is  the 
responsibility  of  the  Milestone  Decision  Authority  (MDA).  If  the  MDA  selects  a 
cost  reimbursement-type  contract  for  the  development  (EMD)  phase,  this 
decision  has  to  be  justified  to  the  Congress.  In  the  justification,  the  MDA  is  to 
explain  why  the  technology  is  so  immature  that  a  cost  reimbursement 
contract  is  necessary  (DoD,  2015a). 

•  Two  (2)  programs  have  yet  to  award  contracts  for  the  EMD  phase.  These  are 
both  pre-MDAP  programs:  NGJ  and  3DELRR. 

Conclusion 

This  research  finds  that  early  competitive  prototyping  has  received  only  partial 
implementation  for  pre-MDAP  and  MDAP  programs.  However,  when  implemented,  early 
competitive  prototyping  in  the  TMRR  phase  does  reduce  technology  and  cost  risk  going  into 
the  EMD  phase  and  does  reduce  Program  Acquisition  Unit  Cost  (PAUC)  as  measured  from 
the  development  (EMD)  baseline  estimate. 

Eight  (8)  pre-MDAP  or  MDAP  programs  entering  EMD  after  May  2009  have  received 
waivers  from  competitive  prototyping  prior  to  Milestone  B.  The  most  common  reason  for 
these  waivers  was  that  the  cost  of  early  prototyping  exceeded  the  life-cycle  benefits  in 
constant  dollars.  While  the  GAO  assessed  that  the  rationale  behind  each  of  these  waivers 
was  consistent  with  the  WSARA  of  2009,  deficiencies  in  the  supporting  business  case 
analyses  were  exposed.  The  Defense  Acquisition  Workforce  would  benefit  from  training  in 
the  identification  of  viable  prototyping  scenarios  and  how  to  estimate  the  life-cycle  benefits 
(in  dollars)  that  can  accrue  from  prototyping. 

Thirteen  (13)  pre-MDAP  or  MDAP  programs  entering  EMD  after  May  2009 
conducted  competitive  prototyping  prior  to  Milestone  B.  These  programs  are  showing  more 
reduction  in  PAUC  compared  with  the  programs  for  which  prototyping  was  waived  or  the 
programs  that  did  not  use  early  prototyping.  In  addition,  the  fact  that  many  of  these 
programs  were  able  to  use  a  fixed  price-type  contract  in  their  EMD  phase  points  to  a 
reduction  in  technology  and  cost  risk  as  a  result  of  competitive  prototyping. 
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Appendix  A.  Waivers 

1 .  Armored  Multi-Purpose  Vehicle  (AMPV)  Program.  The  Army’s  AMPV  fleet 
is  to  replace  the  Ml  13  family  of  vehicles.  The  AMPV  will  come  in  five 
variants:  general  purpose,  medical  evacuation,  medical  treatment,  mortar 
carrier  and  mission  command.  The  AMPV  program  acquisition  strategy  is 
based  on  modifying  an  existing  and  operationally  proven  military  vehicle  and 
plans  to  bypass  technology  development  and  begin  in  system  development. 
The  Army  has  also  modified  or  eliminated  some  AMPV  requirements  to 
ensure  that  no  technology  development  is  needed  for  the  AMPV  (GAO, 
2014c). 

o  Costs  Estimated  for  Prototyping  Alternatives.  (1 )  One  contractor 
producing  5  variants  ($198M);  (2)  Two  contractors  producing  5 
variants  each  and  1  Live  Fire  Test  &  Evaluation  vehicle  each  ($341 M). 

o  Benefits  Estimated.  (1)  None  ($0);  (2)  None  ($0). 

o  Schedule  Impact.  (1)19  months;  (2)  31  months. 

o  Type  Contract  for  TMRR  phase.  None. 

o  Type  Contract  for  EMD  phase.  Cost  Plus  Incentive  Fee  (CPIF)  with 
70/30  share  ratio. 

o  Percent  change  in  PAUC  from  development  baseline.  +0.2%. 

2.  B-2  Defensive  Management  System  Modernization  (B-2  DMS  Mod) 
Program.  The  B-2  DMS  Mod  will  detect,  identify,  and  locate  enemy  radar 
systems  and  provide  real-time  threat  avoidance,  threat  warning,  and  threat 
situational  awareness  information  to  the  aircrew.  The  program  is  an  analog  to 
digital  upgrade  that  consists  of  three  subsystems  (antenna,  electronic  support 
measures,  and  avionics  and  graphics  processors)  integrated  onto  existing  B- 
2  aircraft.  Entry  into  the  EMD  phase  was  in  late  fiscal  year  2015  (GAO, 
2014b).  The  prime  contractor  has  conducted  competitions  for  three  key 
subsystems — antennas,  processors,  and  electronic  support  measures. 
Competitive  prototyping  was  also  conducted  for  the  antennas  (GAO,  2015a). 

o  Costs  Estimated  for  Prototyping  Alternatives.  (1)  System  prototyping 
by  a  separate  contractor  from  Northrup  Grumman  ($524. 8M);  (2) 
Subsystem  prototyping  ($28. 5M). 

o  Benefits  Estimated.  (1)  $6.3M;  (2)  $1.3M. 

o  Schedule  Impact.  (1)  Unknown;  (2)  Unknown. 

o  Type  Contract  for  TMRR  phase.  None. 
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o  Type  Contract  for  EMD  phase.  Cost  Reimbursement. 

o  Percent  change  in  PAUC  from  development  baseline.  Unknown. 

3.  Combat  Rescue  Helicopter  (CRH)  Program.  The  Air  Force’s  CRH  program 
will  replace  legacy  HH-60G  Pave  Hawk  helicopters.  The  CRH  will  recover 
personnel  from  hostile  territory,  conduct  civil  search  and  rescue,  provide 
disaster  relief,  and  evacuate  non-combatants.  Learning  from  the  Combat 
Search  and  Rescue  Replacement  Vehicle  program  that  was  canceled  in 
2009,  the  Air  Force  reduced  requirements  for  the  CRH  to  lower  the  cost  and 
ensure  that  no  technology  development  is  needed  (GAO,  2013b). 

o  Costs  Estimated  for  Prototyping  Alternatives.  2  competing  contractor 
prototypes  during  the  EMD  phase  ($725M). 

o  Benefits  Estimated.  Reduction  of  software  and  integration  risk  ($12M). 

o  Schedule  Impact.  Unknown. 

o  Type  Contract  for  TMRR  phase.  None. 

o  Type  Contract  for  EMD  phase.  Fixed  Price  Incentive  (Firm  Target) 
with  50/50  share  ratio. 

o  Percent  change  in  PAUC  from  development  baseline.  +1 .2%. 

4.  Enhanced  Polar  System  (EPS)  Program  Control  and  Planning  Segment 
(CAPS).  EPS  will  provide  extremely  high  frequency,  jam-resistant,  and 
secure  satellite  communications  to  forces  operating  in  the  North  Pole  region. 
EPS  consists  of  two  payloads  hosted  on  classified  satellites,  a  gateway  to 
connect  user  terminals  to  other  communication  systems,  and  a  control  and 
planning  segment  (CAPS)  to  control  the  payloads  and  manage 
communications  (GAO,  2012b). 

o  Costs  Estimated  for  Prototyping  Alternatives.  One  additional  prototype 
and  funding  for  a  second  contractor  through  Preliminary  Design 
Review  ($49M). 

o  Benefits  Estimated.  Negligible  ($0). 

o  Schedule  Impact.  6  to  24  months. 

o  Type  Contract  for  TMRR  phase.  None. 

o  Type  Contract  for  EMD  phase.  Cost  Plus  Incentive  Fee  (CPIF)  with 
50/50  share  ratio. 

o  Percent  change  in  PAUC  from  development  baseline.  -0.7%. 

5.  F-15  Eagle  Passive/Active  Warning  and  Survivability  System  (F-15 
EPAWSS)  Program.  The  program  will  replace  and  upgrade  internal  self¬ 
protection  electronic  warfare  systems  on  fielded  F-15C/E  aircraft.  EPAWSS 
consists  of  four  major  subsystems:  radar  warning  receiver,  electronic 
countermeasure  processer,  countermeasure  dispenser  system,  and  fiber¬ 
optic  towed  decoy.  These  subsystem  capabilities  will  be  fielded  in  two 
increments.  The  program  entered  the  TMRR  phase  in  August  2015  (GAO, 
2015b). 

o  Costs  Estimated  for  Prototyping  Alternatives.  (1)  Competitive 
prototypes  by  2  contractors  ($116.3M);  (2)  Single  prototype  by  one 
contractor  ($38. 3M);  Critical  subsystem  prototyping  by  one  contractor 
($36. 3M). 

o  Benefits  Estimated.  (1)  $7.2M;  (2)  7.2M;  (3)  6.5M. 
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o  Schedule  Impact.  (1)  41  months;  (2)  26  months;  (3)  21  months 
o  Type  Contract  for  TMRR  phase.  Unknown, 
o  Type  Contract  for  EMD  phase.  Unknown, 
o  Percent  change  in  PAUC  from  development  baseline.  Unknown. 

6.  Indirect  Fire  Protection  Capability  Increment  2,  Block  1  (IFPC  Inc  2  Blk 
1)  Program.  IFPC  Increment  2  will  detect,  assess,  and  defend  against 
threats  from  rockets,  artillery,  mortars,  cruise  missiles,  and  unmanned 
aircraft.  Block  1  counters  cruise  missiles  and  unmanned  aircraft  (GAO, 
2014d). 

o  Costs  Estimated  for  Prototyping  Alternatives.  (1 )  Two  competing 
contractor  prototypes  ($208. 5M);  (2)  Two  in-house  prototypes 
($219. 6M). 

o  Benefits  Estimated.  (1)  $9.8M;  (2)  $9.8M. 
o  Schedule  Impact.  (1)  24  months;  (2)  24  months, 
o  Type  Contract  for  TMRR  phase.  Unknown, 
o  Type  Contract  for  EMD  phase.  Unknown, 
o  Percent  change  in  PAUC  from  development  baseline.  Unknown. 

7.  Fleet  Replenishment  Oiler  (T-AO[X])  Program.  Fleet  oilers  transfer  fuel  to 
Navy  surface  ships  that  are  operating  at  sea.  This  non-developmental  and 
commercial-based  program  replaces  the  Navy’s  15  legacy  T-AO  187  Class 
Fleet  Replenishment  Oilers.  The  acquisition  strategy  includes  competitive 
contract  awards  for  industry  studies,  detailed  design  and  construction,  and 
follow-on  ship  procurement  (GAO,  201 4e).  On  5  April  2013,  the  MDA  signed 
the  program’s  Milestone  B  ADM  citing  no  technology  development  required 
and  low  programmatic  risk  (Compendium,  2015). 

o  Costs  Estimated  for  Prototyping  Alternatives.  Two  prototypes 
($1,350M);  One  prototype  ($742M). 
o  Benefits  Estimated.  None  ($370M);  None  ($370M). 
o  Schedule  Impact.  (1)  60  months;  (2)  60  months, 
o  Type  Contract  for  TMRR  phase.  None, 
o  Type  Contract  for  EMD  phase.  Unknown, 
o  Percent  change  in  PAUC  from  development  baseline.  Unknown. 

8.  Presidential  Helicopter  Replacement  (VH-92A)  Program.  The  Navy’s  VH- 
92A  helicopter  will  transport  the  President,  Vice  President,  and  heads  of 
state.  The  program  provides  replacements  for  legacy  VH-3D  and  VH-60N 
helicopters.  Compared  with  the  VH-71  program  that  was  canceled  in  2009, 
the  VH-92A  has  reduced  performance  requirements.  The  acquisition 
approach  is  to  integrate  mature  communications  and  mission  systems  into  an 
existing  commercial  or  military  helicopter.  Since  programmatic  and 
technology  risks  have  been  reduced,  there  is  no  TMRR  phase  (GAO,  2013c). 

o  Costs  Estimated  for  Prototyping  Alternatives.  (1 )  One  contractor 
producing  system  and  subsystem  prototypes  ($782M);  (2)  Two 
contractors  producing  system  and  subsystem  prototypes  ($3,380M). 
o  Benefits  Estimated.  (1)  $0;  (2)  $542M. 
o  Schedule  Impact.  (1)16  months;  (2)  16  months, 
o  Type  Contract  for  TMRR  phase.  None. 
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o 


Type  Contract  for  EMD  phase.  Fixed  Price  Incentive  (Firm  Target) 
with  50/50  share  ratio. 

Percent  change  in  PAUC  from  development  baseline.  -0.5%. 


o 


Appendix  B.  No  Prototyping 

1.  B61  Mod  12  LEP  TKA.  The  objective  of  the  Air  Force’s  B61  Modification  12 


Life  Extension  Program  (LEP)  is  to  upgrade  and  improve  the  accuracy  of  the 
tactical  nuclear  weapons  stockpile.  The  National  Nuclear  Security 
Administration  (Department  of  Energy)  is  responsible  for  the  B61  Mod  12 
LEP  warhead  design,  manufacture  and  assembly.  The  Air  Force  is 
responsible  for  the  tail  kit  assembly  (TKA)  for  use  on  aircraft,  including  the  F- 
35  JSF  and  the  B-21  Long-Range  Strike  Bomber.  The  MDA  directed  that  the 
program  move  directly  into  the  EMD  phase,  bypassing  the  TMRR  phase  and 
any  early  competitive  prototyping.  The  first  35-month  phase  of  a  total  74- 
month  EMD  contract  was  awarded  to  Boeing  in  November  2012.  Preliminary 
Design  Review  (PDR)  was  planned  for  January  2014,  with  Critical  Design 
Review  (CDR)  in  the  summer  of  2015.  The  development  contract  includes  the 
requirement  to  build  77  engineering  design  models  for  testing  and  evaluation, 
o  Competitive  Prototyping  Tests.  None  (TMRR  phase  bypassed), 
o  TRL/MRL  achieved  through  competitive  prototyping.  Not  applicable, 
o  Type  Contract  for  TMRR  phase.  No  TMRR  phase  or  contract, 
o  Type  Contract  for  EMD  phase.  CPIF  with  30/70  share  ratio, 
o  Percent  change  in  PAUC  from  development  baseline.  -9.6%. 

2.  F-22  Inc  3.2B  Mod.  With  this  modification,  the  Air  Force  is  improving  the  F- 
22  Raptor  hardware  and  software  with  air-to-air  missile  upgrades  (AIM-9X 
and  AIM-1 20D),  Geolocation  2,  and  other  Electronic  Protection  capabilities. 
The  modernization  effort  started  as  an  engineering  change  proposal  (ECP). 
However,  early  development  and  production  cost  estimates  revealed  that  the 
ECP  should  become  a  separate  ACAT  ID  acquisition  program  (DoD  SE 
FY12  Annual  Report).  The  Materiel  Development  Decision  (MDD)  for  the  F- 
22  Inc  3.2B  Mod  was  conducted  in  October  201 1 .  The  February  201 2  R-2 
Budget  Exhibits  for  the  program  mention  the  use  of  an  Avionics  Integration 
Lab  (AIL),  but  no  prototyping.  The  program  achieved  Milestone  B  in  June 
2013. 

o  Competitive  Prototyping  Tests.  None.  No  prototypes  were  built, 
o  TRL/MRL  achieved  through  competitive  prototyping.  Not  applicable, 
o  Type  Contract  for  TMRR  phase.  IDIQ 

o  Type  Contract  for  EMD  phase.  CPIF  with  20/80  share  ratio  for  EMD; 

CPFF  for  test  execution. 

o  Percent  change  in  PAUC  from  development  baseline.  -2.3%. 

3.  FAB-T-FET.  The  Air  Force’s  Family  of  Advanced  Beyond  Line-of-Sight 
Terminals  (FAB-T)  provides  for  survivable  terminals  for  communicating 
strategic  nuclear  execution  orders  via  jam-resistant,  low  probability  of 
intercept  waveforms  through  the  Milstar  and  Advanced  Extremely  High 
Frequency  (AEHF)  satellite  constellations.  FAB-T  includes  both  Command 
Post  Terminals  (CPT)  and  Force  Element  Terminals  (FET).  On  30  July  2015, 
Congress  was  notified  by  the  USD(AT&L)  that  the  FAB-T  program  had  been 
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split  into  two  subprograms  for  more  effective  management:  CPT  and  FET. 

The  FAB-T-CPT  subprogram  entered  the  Production  and  Deployment  phase 
in  October  2015.  However,  execution  of  the  FAB-T-FET  subprogram  has 
been  deferred,  in  part  due  to  bomber  aircraft  requirements  still  under 
development  by  Air  Force  Global  Strike  Command  and  approval  of  the 
Acquisition  Program  Baseline  (APB;  DAES,  2016a). 

o  Competitive  Prototyping  Tests.  None.  No  prototypes  have  been  built, 
o  TRL/MRL  achieved  through  competitive  prototyping.  Not  applicable, 
o  Type  Contract  for  TMRR  phase.  Unknown, 
o  Type  Contract  for  EMD  phase.  Not  awarded  yet. 
o  Percent  change  in  PAUC  from  development  baseline.  +4.1%. 

4.  GPS  OCX.  The  Air  Force’s  Global  Positioning  System  (GPS)  OCX  program 
provides  for  a  modernized  satellite  command  and  control  (C2)  system  which 
replaces  the  current  ground  control  system  for  legacy  and  new  GPS 
satellites.  OCX  implements  a  modern  flexible  architecture  with  information 
assurance  built  in  to  address  emerging  cyber  threats.  The  Air  Force  is  taking 
a  block  approach  to  develop  OCX,  with  each  block  delivering  upgrades  as 
they  become  available  (DAES,  2016b). 

o  Competitive  Prototyping  Tests.  None.  No  prototypes  have  been  built, 
o  TRL/MRL  achieved  through  competitive  prototyping.  Not  applicable, 
o  Type  Contract  for  TMRR  phase. 

o  Type  Contract  for  EMD  phase.  Cost  Plus  Award  Fee  (CPAF) 
o  Percent  change  in  PAUC  from  development  baseline.  +21.2%. 

5.  ICBM  Fuze  Mod.  The  Air  Force’s  Intercontinental  Ballistic  Missile  (ICBM) 
Fuze  Modernization  (Fuze  Mod)  program  entails  the  design  and  development 
of  a  form,  fit,  and  function  replacement  for  the  current  Mk21  fuze  used  on  the 
W87  warhead.  This  is  a  cooperative  effort  between  the  Department  of 
Energy’s  National  Nuclear  Security  Administration  (NNSA),  the  U.S.  Navy, 
and  the  United  Kingdom.  The  ICBM  Fuze  Mod  program  will  leverage 
technologies,  components,  and  development/production  capabilities  from 
previous  work  performed  by  the  U.S.  Navy  and  NNSA  on  the  Mk5  fuze  for 
Submarine  Launched  Ballistic  Missile  warheads.  The  Mk  5  fuze  entered  EMD 
in  August  2012.  The  ICBM  Fuze  Mod  program  entered  the  EMD  phase  in 
August  2013  after  6.3  Development  Engineering  entry  approval  by  the 
Nuclear  Weapons  Council  (chaired  by  the  USD[AT&L]).  Entry  was  in 
accordance  with  DOE  Instruction  5030.55.  The  equivalent  milestone  review  in 
DoDI  5000.02  is  Milestone  B  (R-2  Budget  Exhibit,  2014). 

o  Competitive  Prototyping  Tests.  None.  No  prototypes  have  been  built, 
o  TRL/MRL  achieved  through  competitive  prototyping.  Not  applicable, 
o  Type  Contract  for  TMRR  phase.  None.  The  TMRR  phase  was 
bypassed. 

o  Type  Contract  for  EMD  phase.  Sandia  National  Labs,  Albuquerque, 
NM,  MIPR;  Northrup  Grumman,  Clearfield,  CT,  CPAF 

o  Percent  change  in  PAUC  from  development  baseline.  +1 .0%. 

6.  KC-46A.  The  Air  Force’s  Tanker  Modernization  (KC-46A)  program  will 
replace  the  aging  fleet  of  air  refueling  tankers.  Since  the  program  is  based 
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upon  a  commercial  aircraft  modified  for  air  refueling  and  military  avionics,  the 
program  bypassed  the  TMRR  phase  and  any  competitive  prototyping.  An 
EMD  phase  development  contract  was  awarded  to  Boeing  in  February  201 1 . 
A  successful  PDR  was  conducted  in  April  2013.  A  Milestone  C  decision  is 
expected  in  April  2016  (DAES,  2016a). 

o  Competitive  Prototyping  Tests.  None.  No  competitive  prototypes  were 
built. 

o  TRL/MRL  achieved  through  competitive  prototyping.  Not  applicable, 
o  Type  Contract  for  TMRR  phase.  None.  TMRR  phase  was  bypassed, 
o  Type  Contract  for  EMD  phase.  Fixed  Price  Incentive  (Firm  Target; 
FPIF)  with  a  40/60  share  ratio. 

o  Percent  change  in  PAUC  from  development  baseline.  -8.2%. 

7.  LCS  MM.  The  Mission  Modules  (MM)  for  the  Navy’s  Littoral  Combat  Ship 
(LCS)  provide  Mine  Countermeasures  (MCM),  Surface  Warfare  (SUW),  and 
Anti-Submarine  Warfare  (ASW)  capabilities.  While  EMD  phase  prototypes 
are  currently  in  development  and  testing,  early  MM  prototyping  in  the  TMRR 
phase  is  unknown.  This  could  be  because  the  MM  became  reportable  as  an 
MDAP  only  when  all  three  MM  were  combined  into  the  single  LCS  MM 
program.  However,  the  LCS  MM  program  office  feels  that  it  is  complying  with 
DoD  guidance  and  regulations  because  all  MM  were  demonstrated  in  a 
relevant  environment  prior  to  MM  integration  with  the  LCS  (GAO,  2015a). 
o  Competitive  Prototyping  Tests.  None.  No  early  LCS  MM  prototypes 
were  tested  on  the  actual  LCS. 

o  TRL/MRL  achieved  through  competitive  prototyping.  Not  applicable, 
o  Type  contract  for  TMRR  phase.  Unknown, 
o  Type  Contract  for  EMD  phase.  CPFF  and  CPAF 
o  Percent  change  in  PAUC  from  development  baseline.  -0.1%. 

Appendix  C.  Prototyping 

1 .  AMDR.  The  Navy’s  next  generation  radar  system  for  air  defense  and  ballistic 
missile  defense  is  the  Air  and  Missile  Defense  Radar  (AMDR).  Using  S-band 
radar,  C-band  radar,  and  a  Radar  Suite  Controller,  the  AMDR  will  be 
deployed  on  Guided  Missile  Destroyer  (DDG)  51  Flight  III.  TMRR  phase 
prototypes  of  these  AMDR  components  were  developed  by  Lockheed  Martin, 
Northrup  Grumman,  and  Raytheon  based  upon  contracts  awarded  in 
September  2010.  An  initial  PDR  was  conducted  prior  to  MS  B  with  each  of 
the  three  TMRR  phase  solutions.  However,  a  system-level  Preliminary 
Design  Review  (PDR)  for  the  AMDR  was  not  conducted  until  27  August  2014 
under  the  EMD  contract  awarded  to  Raytheon.  The  MDA  had  waived  the 
requirement  to  conduct  a  system-level  PDR  prior  to  Milestone  B  (AMDR 
DAES,  2016). 

o  Competitive  Prototyping  Tests.  Hardware  components  were 

prototyped  by  three  contractors.  Software  was  not  prototyped  (DoD 
SE  FY2013  Annual  Report). 

o  TRL/MRL  achieved  through  competitive  prototyping.  The  TMRR 

phase  demonstrated  the  critical  technologies  necessary  for  a  scalable 
AMDR  (DoD  SE  FY201 3  Annual  Report). 
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o 


Type  Contract  for  TMRR  phase.  3  FPIF  contracts  for  $85.392M  each 
and  3  FFP  contracts  for  $10M  each  to  LM,  NG,  and  Raytheon  (R-3 
Budget  Exhibit,  2011). 

o  Type  Contract  for  EMD  phase.  CPIF  with  50/50  share  ratio 
o  Percent  change  in  PAUC  from  development  baseline.  -10.5%. 

2.  Chem-Demil  ACWA.  DoD’s  Chemical  Demilitarization-Assembled  Chemical 
Weapons  Alternatives  (ACWA)  program  includes  two  chemical 
demilitarization  plants  designed  to  safely  dispose  of  the  chemical  weapons 
stored  at  Blue  Grass  and  Pueblo  Army  Depots.  Instead  of  incineration,  both 
plants  use  safe  and  environmentally  sound  neutralization  alternatives.  The 
program  suffered  a  critical  unit  cost  breach  in  2010  and  was  rebaselined  to 
cover  risk  and  to  prove  out  first-of-a-kind  equipment  development  (Annual 
SAR,  2010).  Since  then,  event-based  technical  reviews  of  a  rocket  cutting 
machine  and  a  rocket  shearing  machine  were  conducted  prior  to  the  March 
2012  Milestone  B  decision.  Both  machines  were  found  ready  for 
implementation.  In  addition,  each  plant  was  modeled  to  assist  in 
understanding  facility  control  system  issues  (DoD  SE,  FY12  Annual  Report). 

o  Competitive  Prototyping  Tests.  Component  testing  only  as  discussed 
above. 

o  TRL/MRL  achieved  through  competitive  prototyping.  Not  available, 
o  Type  Contract  for  TMRR  phase.  Bluegrass,  CPIF  (awarded  13  June 
2003);  Pueblo,  CPAF  (awarded  27  September  2002) 
o  Type  Contract  for  EMD  phase.  Contracts  from  TMRR  phase  have 
been  extended. 

o  Percent  change  in  PAUC  from  development  baseline.  +3.0%. 

3.  EPS.  The  Air  Force’s  Enhanced  Polar  System  (EPS)  program  will  provide 
assured  communications  over  the  North  Polar  Region.  The  system  consists 
of  two  communications  payloads  on  host  satellites  in  highly  elliptical  orbits. 
The  EPS  is  composed  of  and  managed  in  four  segments:  an  extended  data 
rate  payload,  user  terminals  (acquired  separately  by  users),  a  fixed 
installation  Gateway,  and  a  fixed  installation  Control  and  Planning  Segment 
(OUSD[AT&L],  n.d.).  The  EPS  payloads  are  a  simplification  of  the  Advanced 
Extremely  High  Frequency  (AEHF)  payloads,  and  the  EPS  segments  use  the 
mature  extended  data  rate  waveform  with  common  Global  Information  Grid 
interfaces.  Since  technology  was  determined  to  be  mature,  the  EPS  program 
was  directed  in  2007  to  proceed  to  Milestone  B.  Before  that  review,  the  EPS 
program  office  and  contractor  did  analysis  and  modeling  and  conducted  a 
Preliminary  Design  Review  (PDR)  at  which  an  allocated  baseline  was 
established.  The  DoD  Director  of  Systems  Engineering  assessed  that 
allocated  baseline  for  software  development  and  determined  that  it  lacked 
integration  detail  (DoD  SE,  FY13  Annual  Report).  Regardless,  a  successful 
Milestone  B  decision  review  was  conducted  on  2  April  2014.  Today,  the  prime 
contractor,  Northrup  Grumman,  is  experiencing  some  unfavorable  cost 
variance  due  to  unplanned  software  and  systems  engineering,  integration, 
and  testing;  however,  there  are  no  significant  software  development  issues 
(DAES,  2016a). 
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o  Competitive  Prototyping  Tests.  Early  prototyping  included  flight 

equivalent  payloads,  a  gateway  engineering  development  model,  and 
prototype  control  and  planning  software  (GAO,  2015a). 

o  TRL/MRL  achieved  through  competitive  prototyping.  Not  available, 
o  Type  Contract  for  TMRR  phase.  Cost  Plus  Incentive  Fee  (CPIF) 
o  Type  Contract  for  EMD  phase.  CPIF  with  a  50/50  share  ratio, 
o  Percent  change  in  PAUC  from  development  baseline.  -0.6%. 

4.  F-35  Aircraft.  The  Joint  Strike  Fighter  (F-35  Aircraft)  program  develops, 
produces,  and  fields  the  next  generation  multi-role  tactical  aircraft.  The  three 
variants  are  F-35A  Conventional  Takeoff  and  Landing  (CTOL)  for  the  Air 
Force,  F-35B  Short  Takeoff  and  Vertical  Landing  (STOVL)  for  the  Marine 
Corps,  and  F-35C  Aircraft  Carrier  suitable  Variant  (CV)  for  the  Navy  (DAES, 
2016b). 

o  Competitive  Prototyping  Tests.  Two  competitive  prototypes  were  built, 
o  TRL/MRL  achieved  through  competitive  prototyping.  Not  available, 
o  Type  Contract  for  TMRR  phase.  CPAF/CPIF 
o  Type  Contract  for  EMD  phase.  CPAF/CPIF 
o  Percent  change  in  PAUC  from  development  baseline.  -5.3%. 

5.  F-35  Engine.  The  Joint  Strike  Fighter  Engine  (F-35  Engine)  program  involved 
competition  between  two  contractors:  General  Electric  and  Pratt  and  Whitney. 
Due  to  cost,  the  DoD  decided  to  go  with  just  one  contractor,  Pratt  and 
Whitney. 

o  Competitive  Prototyping  Tests.  Prototypes  were  built  by  2  different 
contractors  (Pratt  and  Whitney  and  General  Electric), 
o  TRL/MRL  achieved  through  competitive  prototyping.  Not  available, 
o  Type  Contract  for  TMRR  phase.  CPAF/CPIF 
o  Type  Contract  for  EMD  phase.  CPAF/CPIF 
o  Percent  change  in  PAUC  from  development  baseline.  -4.7%. 

6.  JAGM.  The  Joint  Air  Ground  Missile  (JAGM)  program  is  led  by  the  Army  with 
joint  interest  with  the  U.S.  Marine  Corps  and  U.S.  Navy.  The  JAGM 
represents  the  next  generation  of  aviation  launched  fire  and  forget  missiles  to 
replace  the  HELLFIRE  laser  and  Longbow  radar  missiles.  JAGM  will  be  used 
by  joint  service  aircraft  for  destruction  of  high  value  stationary,  moving,  and 
relocatable  land  and  maritime  targets  from  standoff  range  in  day,  night, 
adverse  weather,  and  obscured  battlefield  conditions  (DAES,  2016b). 

o  Competitive  Prototyping  Tests.  Two  contractors  built  prototypes  to 
mature  technologies  and  designs.  Each  contractor  completed  a  flight 
test  and  a  Preliminary  Design  Review  (PDR).  The  program’s  four 
critical  technologies  were  all  approaching  maturity  before  the  EMD 
phase  (GAO,  2015a). 

o  TRL/MRL  achieved  through  competitive  prototyping.  Not  available, 
o  Type  Contract  for  TMRR  phase.  Unknown, 
o  Type  Contract  for  EMD  phase.  Not  yet  awarded  (FFP  contract 
planned). 

o  Percent  change  in  PAUC  from  development  baseline.  +0.2%. 
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7.  JLTV.  The  DoD’s  Joint  Lightweight  Tactical  Vehicle  (JLTV)  Program  was  one 
of  the  first  programs  to  comply  with  competitive  prototyping  in  the  TMRR 
phase  (at  the  time,  competitive  prototyping  was  a  USD[AT&L]  policy,  not  a 
statutory  requirement).  In  October  2008,  three  separate  Cost  Plus  Fixed  Fee 
(CPFF)  contracts,  totaling  $239.8  million,  were  awarded  for  the  design, 
development,  modeling,  simulation,  fabrication,  test,  and  test  support  of  24 
JLTVs  and  companion  trailers  (JLTV  DAES  ExSum,  2016).  The  three  vendor 
prototypes  were  subjected  to  endurance  testing  to  demonstrate  reliability  and 
maintainability,  user  assessments  of  suitability,  and  ballistic  testing  to  assess 
force  protection  requirements  (DOT&E,  2011). 

o  Competitive  Prototyping  Tests.  During  TMRR  phase  testing,  the 
DOT&E  reported  that  all  three  vendor  prototypes  had  problems  with 
mobility  in  soft  soil  and  integrating  government  furnished  mission 
equipment  into  the  prototypes.  Reliability  was  also  an  issue  for  all 
prototypes,  falling  well  short  of  the  threshold  reliability  of  2,400  mean 
miles  between  operational  mission  failure.  In  addition,  during  the 
TMRR  phase,  the  JLTV  underbody  threat  requirement  was  increased 
to  that  of  the  Mine  Resistant  Ambush  Protected  (MRAP)  vehicle.  As  a 
result,  this  increased  force  protection  requirement  was  not  tested  on 
the  JLTV  competitive  prototypes  (DOT&E,  2011;  DOT&E,  2012). 
o  TRL/MRL  achieved  through  Competitive  Prototyping.  Not  available, 
o  Type  Contract  for  TMRR  phase.  Cost  Plus  Fixed  Fee  (CPFF)/Cost 
Share 

o  Type  Contract  for  EMD  phase.  Three  Firm  Fixed  Price  (FFP) 

contracts,  with  a  14-month  period  of  performance,  for  22  vehicles  and 
six  trailers  from  each  vendor  (JLTV  DAES  ExSum,  2016). 

o  Percent  change  in  PAUC  from  development  baseline.  -16.2%. 

8.  LCS.  The  Navy’s  Littoral  Combat  Ship  (LCS)  is  optimized  for  flexibility  in  the 
littorals  as  a  system  of  systems,  both  manned  and  unmanned.  The  LCS  will 
be  reconfigurable  with  three  mission  packages:  surface  warfare,  mine 
warfare,  and  littoral  anti-submarine  warfare  (DAES,  2016b). 

o  Competitive  Prototyping  Tests.  Two  competitive  prototypes  were  built, 
each  of  a  different  design,  by  separate  contractors, 
o  TRL/MRL  achieved  through  competitive  prototyping.  Not  available, 
o  Type  Contract  for  TMRR  phase.  FPIF 
o  Type  Contract  for  EMD  phase.  FPIF 

o  Percent  change  in  PAUC  from  development  baseline.  +2.7%. 

9.  NGJ.  The  Navy’s  Next  Generation  Jammer  (NGJ)  is  an  electronic  attack 
system  that  will  provide  significantly  improved  Airborne  Electronic  Attack 
capabilities  against  advanced  threats  through  enhanced  agility  and  precision 
jamming,  increased  interoperability,  and  greater  coverage  against  a  wide 
variety  of  radio  frequency  emitters.  The  NGJ  system  will  be  integrated  on  the 
EA-18G  tactical  aircraft  to  replace  aging  AN/ALG-99  Tactical  Jamming 
System  (TJS)  pods  (DAES,  2016b). 

o  Competitive  Prototyping  Tests.  Four  prototypes  have  been  built  and 
tested:  flight  demonstration  pod,  8%  scale  model  in  a  high  speed  wind 
tunnel,  and  component-level  prototypes  fortesting  of  the  Mid-Band  1 
and  Mid-Band  2  array  subsystem. 
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o  TRL/MRL  achieved  through  competitive  prototyping.  Not  available, 
o  Type  Contract  for  TMRR  phase.  Unknown, 
o  Type  Contract  for  EMD  phase.  Pre-MDAP,  not  awarded  yet. 
o  Percent  change  in  PAUC  from  development  baseline.  Unknown.  The 
NGJ  development  baseline  cost  estimate  is  not  known  yet. 

10.  SDB  II.  The  Small  Diameter  Bomb  II  (SDB  II)  program  is  a  joint  program  with 
the  Air  Force  as  the  lead  service.  The  SDB  II  is  a  250  pound,  air-launched, 
precision-glide  weapon  that  can  attack  both  stationary  and  moving  targets  in 
degraded  weather  conditions  at  standoff  ranges.  The  SDB  II  uses  millimeter- 
wave  radar,  infrared  sensor,  and  semi-active  laser  technologies,  in  addition  to 
GPS  and  inertial  navigation.  The  SDB  II  also  has  a  weapon  datalink  network 
for  in-flight  target  updates,  retargeting,  weapon  tracking,  and  weapon  abort 
(DOT&E,  FY15  Annual  Report).  The  SDB  II  program  entered  the  TMRR 
phase  in  2006.  Competitive  prototype  contracts  (CPFF)  were  awarded  to 
Boeing  and  Raytheon  in  May  2006.  By  October  2009,  SDB  II  had  completed 
a  42-month  competitive  TMRR  phase.  The  program  had  a  successful  MS  B 
review  and  entered  the  EMD  phase  on  29  July  2010  (R-2  Budget  Exhibits, 
2007,2010,2011). 

o  Competitive  Prototyping  Tests:  Both  contractor  prototypes  were 

subjected  to  free  flight  demonstrations,  captive  carriage  on  F-15s,  and 
seeker  testing  at  the  component  level.  Each  contractor’s  warhead  was 
also  tested  for  lethality  (Director,  OT&E  FY09  Annual  Report), 
o  TRL/MRL  achieved  through  competitive  prototyping.  Competitive 
prototyping  lead  to  a  near-CDR  level  of  maturity  prior  to  the  down- 
select  and  MS  B  decision.  One  or  both  contractors  were  on  track  to 
achieve  TRL  6  or  greater  and  MRL  6  or  greater  by  MS  B  (DoD  SE 
FY09  Annual  Report). 

o  Type  Contract  for  TMRR  phase.  Cost  Plus  Fixed  Fee  (CPFF) 

o  Type  Contract  for  EMD  phase.  Fixed  Price  Incentive  (Firm  Target; 
FPIF)  with  priced  options  for  the  first  5  production  lots  (FPIF,  lots  1-3 
and  FP[EPA],  lots  4  and  5). 

o  Percent  change  in  PAUC  from  development  baseline.  -21 .3%. 

1 1 .  SF  Inc  1.  The  Air  Force’s  Space  Fence  (SF)  Ground  Based  Radar  System 
Increment  1  (Inc  1)  will  provide  space  situational  awareness  by  detecting  and 
reporting  objects  in  Low  Earth  and  Medium  Earth  Orbits  (LEO/MEO).  The 
system  will  have  an  operations  center  and  two  radar  sites  operating  at  S- 
band  frequencies.  SF  Inc  1  consists  of  the  operations  center  at  Reagan  Test 
Site  Operations  Center  Huntsville,  AL,  and  one  radar  site  located  at  Kwajalein 
Atoll,  Republic  of  the  Marshall  Islands  (OUSD[AT&Lj,  n.d.).  The  program  had 
a  successful  MS  B  in  3rd  Qtr  FY2014  and  is  now  in  the  EMD  phase. 
Negotiations  are  still  underway  with  the  potential  host  nation  for  the  location 
for  the  second  radar  site  (R-2,  PE  0604425F,  Project  65A009,  February 
2016). 

o  Competitive  Prototyping  Tests.  Two  working  prototypes  were  built  in 
the  TMRR  phase  to  reduce  risk.  Also  used  prototype  assembly  to 
validate  production  planning  for  IOC  (DoD  SE  FY2012  and  2013 
Annual  Reports). 
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o  TRL/MRL  achieved  through  competitive  prototyping.  An  Independent 
Program  Assessment  (IPA)  in  FY2012  assessed  the  two  contractor 
designs  as  technically  mature  and  validated  by  working  prototypes 
(DoD  SE  FY201 2  Annual  Report), 
o  Type  Contract  for  TMRR  phase.  Firm  Fixed  Price  (FFP) 
o  Type  Contract  for  EMD  phase.  Fixed  Price  Incentive  (Firm  Target) 
with  a  20/80  share  ratio. 

o  Percent  change  in  PAUC  from  development  baseline.  -4.9%. 

12.  SSC.  The  Ship  to  Shore  Connector  (SSC)  is  the  Navy’s  replacement  for  the 
Landing  Craft,  Air  Cushion  (LCAC)  class  of  ships.  The  SSC  will  project, 
sustain,  and  retrograde  combat  power  from  the  sea,  independent  of  tides, 
water  depth,  underwater  obstacles,  or  beach  gradient.  The  lead  craft  (funded 
with  RDT&E,  Navy)  will  be  maintained  as  a  test  and  training  craft  to  test  fixes 
to  problems  that  arise  during  fleet  introduction  (R-2  Budget  Exhibit,  2010). 
The  SSC  received  Milestone  B  approval  in  June  2012.  In  July  2015,  the 
program  was  granted  approval  to  enter  the  Production  and  Deployment 
phase. 

o  Competitive  Prototyping  Tests.  The  Command  Module  (a  subsystem) 
was  prototyped  (DoD  SE  FY2014  Annual  Report), 
o  TRL/MRL  achieved  through  competitive  prototyping.  Not  available, 
o  Type  Contract  for  TMRR  phase.  Cost  Plus  Fixed  Fee  (CPFF)  and 
Firm  Fixed  Price  (FFP) 

o  Type  Contract  for  EMD  phase.  Fixed  Price  Incentive,  Firm  Target  (FPI 
[F])  to  Textron,  Inc.  for  detailed  design  and  construction  of  lead  test 
and  training  craft  (Craft  100)  and  technical  manuals, 
o  Percent  change  in  PAUC  from  development  baseline.  -6.7%. 

13.  3DELRR.  The  Air  Force’s  Three  Dimensional  Long-Range  Radar  (3DELRR) 
is  a  replacement  for  the  current  legacy  AN/TPS-75  Radar.  The  3DELRR  is  a 
software  intensive  program  that  uses  new  semiconductor  technology 
(gallium-nitride-based  transmit/receive  modules  that  have  lower  power 
requirements). 

o  Competitive  Prototyping  Tests.  Three  competing  contractors  built  and 
demonstrated  critical  software  elements  and  internal  system 
integration. 

o  TRL/MRL  achieved  through  competitive  prototyping.  Results 
presented  at  the  Preliminary  Design  Reviews  from  Technical 
Performance  Measures  indicate  that  the  program  is  on  track  to  meet 
its  six  KPPs  and  seven  KSAs  (DoD  SE  FY2013  Annual  Report), 
o  Type  Contract  for  TMRR  phase.  Cost  Plus  Incentive  Fee  (CPIF;  1 
contractor)  and  Firm  Fixed  Price  (FFP;  2  contractors), 
o  Type  Contract  for  EMD  phase.  Fixed  Price  Incentive  Firm  Target 
(FPIF)  planned. 

o  Percent  change  in  PAUC  from  development  baseline.  Unknown.  The 
3DELRR  development  baseline  cost  estimate  is  not  known  yet. 
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WSARA  and  Prototyping 

•  Competitive  prototyping  of  systems  o£ajtico[ 
subsystems  before  Milestone  B  approval 

•  If  competitive  prototyping  is  waived  by  MDA,  a 
prototype  must  still  be  produced  before  MS  B 
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•  Rationale  behind  waivers 

•  GAO's  Assessments: 

-  Some  waivers  supported  by  Acquisition  Strategy 
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-  Schedule  impacts  not  quantified  in  dollars 
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Rethink  and  Relearn  Prototyping 

•  How  to  develop  a  sound  business  case 

•  Multiple  competition-sensitive  contracts 

•  Defending  cost  of  multiple  prototypes 

•  Understanding  how  ATDs  and  JCTDs  fit  in 

•  Competitive  prototyping  of  critical  subsystems 
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